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Workshop  Session  #4: 

Learning  and  Human  Effectiveness  in  Embedded  Virtual  Simulations 

Summary  of  Discussion 


This  session  was  facilitated  by  Ms.  Claudy  Koerhuis  (NED).  Student  modeling,  coaching  strategies  and 
tutor  perception  and  their  relationship  to  EVS  were  discussed.  Four  papers  were  presented  as  noted 
below: 

•  Making  a  case  for  machine  perception  of  trainee  affect  to  aid  learning  and  performance  in 
embedded  virtual  simulations  -  Robert  Sottilare,  Ph.D.  (USA),  U.S.  Army  Research  Development 
&  Engineering  Command  -  Simulation  &  Training  Technology  Center 

•  Intelligent  tutoring  methods  for  optimizing  learning  outcomes  with  embedded  training  -  Randy 
Jensen  (USA),  Stottler-Henke;  Jeffery  Mosley,  Ph.D.  (USA),  Optimetrics,  Inc.;  COL  Michael 
Sanders  (USA),  DoD  Modeling  &  Simulation  Coordination  Office;  and  MAJ  Jason  Sims  (USA), 
U.S.  Army  Command  and  General  Staff  College 

•  Spoken  dialogue:  extending  embedded  virtual  simulation  with  a  very  human  dimension  - 
Benjamin  Bell,  Ph.D.  (USA),  Chi  Systems,  Inc.;  and  Philip  Short  (UK),  Aerosystems  International 
Ltd. 

•  Intelligent  agent- supported  autonomous  training  in  virtual  simulation  environments  -  Annerieke 
Heuvelink,  Ph.D.  (NED),  TNO  Defense  Security  and  Safety;  Karel  van  den  Bosch,  Ph.D.  (NED), 
TNO  Defense  Security  and  Safety;  Willem  A.  van  Doesburg  (NED),  TNO  Defense  Security  and 
Safety  ;  and  Maaike  Harbers  (NED),  TNO  Defense  Security  and  Safety 

Workshop  Exercise  #4:  Mindmapping  learning  in  EVS 

For  this  session,  the  mindmapping  exercise  focused  on  technology  (tools  and  methods)  to  enhance 
learning  in  EVS.  The  questions  below  were  intended  to  focus  and  energize  the  discussion: 

•  What  are  the  factors  enabling/limiting  learning  in  EVS? 

•  When  is  the  optimal  time  to  provide  feedback  in  EVS?  On  error,  on  trainee  request,  other? 

•  How  much  human  involvement  is  required  to  support  learning  in  EVS  and  how  much  can  be 
delegated  to  intelligent  agents? 

•  How  is  feedback  provided  to  trainees  during  embedded  training? 

The  mindmaps  shown  below  are  the  group  products  from  Exercise  #4  on  learning  in  EVS. 
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Figure  1:  Group  1  Mindmap  for  Exercise  #4 
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Figure  2:  Group  2  Mindmap  for  Exercise  #4 
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Figure  3:  Group  3  Mindmapfor  Exercise  #4 

The  three  mindmaps  above  resulted  in  four  common  themes:  feedback  methods,  feedback  frequency, 
automation  and  learning  enablers/limitations.  Each  is  discussed  below: 

Feedback  Methods:  The  groups  identified  different  methods  to  provide  feedback  (e.g.  via  human  instructor 
or  via  intelligent  agents)  and  concluded  that  this  feedback  can  be  both  intrinsic  and  extrinsic. 

The  groups  identified  two  important  points  related  to  feedback  delivery: 
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•  Feedback  should  be  presented  when  trainee’s  actual  performance  varies  sufficiently  from  expected 
performance 

•  Feedback  should  be  adapted  (modified)  based  on  knowledge  of  trainee’s  performance 

Feedback  Frequency:  The  groups  identified  three  important  points  related  to  feedback  frequency: 

•  Feedback  frequency  should  be  task  dependent 

•  Feedback  should  be  triggered  by  mistakes 

•  Feedback  should  be  triggered  by  trainee  requests 


Automation:  The  groups  thought  more  automation  was  needed  in  training  and  reinforced  the  following 
points: 

•  Automation  should  be  provided  in  situations  where  human  tutors  are  either  unavailable  or  that  it  is 
not  practical/cost  effective. 

•  Automation  should  be  provided  for  training  support  functions  to  help  reduce  costs. 

•  Intelligent  agents  require  additional  development  in  the  areas  of  tutors,  scenario  directors, 
coaches,  player  agents  (team  members)  and  process  management  agents 

Learning  Enablers  and  Limitations:  The  following  learning  enablers/limitations  were  identified: 

•  Increased  availability  of  training  through  EVS  as  training  goes  with  the  Warfighter  (enabler). 

•  Limited  availability  of  instructors/support  personnel  for  EVS  (limitation) 

•  Limited  capabilities  of  current  intelligent  agents  (e.g.  tutors)  including  adaptiveness  and  ability  to 
motivate  trainees 

•  The  need  for  more  natural  interfaces/interaction  with  virtual  simulations  (limitation) 

•  Incomplete  understanding  of  trainee  state  based  on  methods  (limitation) 

Each  of  the  limitations  noted  above  are  also  opportunities  for  future  research. 
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